On-line Learning in the Committee Machine

The dynamics of learning from examples in the K = 3 nonoverlapping committee machine with single presentation of examples is studied. The optimal algorithm, in the sense of mean generalization, is obtained from a variational analysis of the diierential equations which describe the dynamics. The agreement of the theoretical predictions and the results of numerical simulations is excellent. The optimized dynamics has the extra advantage with respect to the nonoptimized cases in that it uncouples the differential equations which describe the evolution of the relevant parameters, i.e. the student-teacher overlap and the norm of the student synaptic vector. This in turn translates into the possibility of constructing useful practical optimized on-line algorithms which work optimally even in the absence of knowledge of the probability distribution of examples. For the optimal algorithm the generalization error decays as 0:88 ?1 , the same nominal error as for the simple perceptron with optimized dynamics.